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Abstract. The thermodynamic and retrieval properties of the Blume-Emery-Griffiths neural network with 
synchronous updating and variable dilution are studied using replica mean-field theory. Several forms of 
dilution are allowed by pruning the different types of couplings present in the Hamiltonian. The appearance 
and properties of two-cycles are discussed. Capacity-temperature phase diagrams are derived for several 
values of the pattern activity. The results are compared with those for sequential updating. The effect of 
self-coupling is studied. Furthermore, the optimal combination of dilution parameters giving the largest 
critical capacity is obtained. 
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1 Introduction 

Recently, we have studied the possible different physics 
arising from sequential and synchronous updating of the 
spins in multi-state Ising-typc ferromagnets [Q. In gen- 
eral, it is known that besides fixed-point attractors, syn- 
chronous updating in spin-models can give rise to cycles 
as stationary behaviour |2I3| . In we have shown that 
for the Q = 3 Ising ferromagnet with synchronous up- 
dating the same stationary solutions appear as for se- 
quential updating except for negative couplings. In that 
case, for synchronous updating, cycles of period two in the 
magnetization occur in the ferromagnetic phase. For the 
Blume-Emery-Griffiths (BEG) ferromagnet, however, we 
have found that synchronous updating allows for a much 
richer free energy landscape and phase diagram. 

In view of these results it is of interest to extend such 
a study to multi-state neural networks in order to see 
whether the different types of updating give rise to differ- 
ent retrieval properties. For the Hopfield neural network 
model, e.g., involving binary neurons, it is known that in a 
replica-symmetric treatment the retrieval region is slightly 
larger in the synchronous case 0]. A natural question is 
then whether a similar effect is present in the BEG neural 
network, which is known to optimize the mutual informa- 
tion content for three-state networks |5l6j and, hence, to 
guarantee the best retrieval properties in comparison with 
other three-state networks (see [7] for a review). For se- 
quential updating, the thermodynamic and retrieval prop- 
erties arc studied in detail in the literature for fully con- 
nected architectures |S| and diluted architectures |9I1U| 
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(and references therein), for synchronous updating such a 
study has not yet appeared. 

The purpose of this work is precisely to report on a 
study of the BEG neural network with synchronous up- 
dating and variable dilution. In particular, we discuss the 
thermodynamic and retrieval properties of this model us- 
ing replica symmetric mean-field theory. Several forms 
of dilution are allowed by cutting some of the couplings 
appearing in the Hamiltonian: the couplings multiplying 
the term bilinear in the neurons and/or the couplings re- 
lated to the term biquadratic in the neurons. Capacity- 
temperature phase diagrams are derived for several values 
of the pattern activity. In particular, the appearance and 
properties of cycles are discussed. Apart from these cycles 
it turns out that the asymptotic behaviour is almost iden- 
tical to the one for sequential updating. For some values 
of the (fully-connected) network parameters the retrieval 
region is marginally enhanced. Only the addition of self- 
coupling, not allowed in the sequential model if one wants 
to satisfy detailed balance |3lllj , can enlarge the retrieval 
region substantially, especially in the case of dilution. The 
optimal combination of the network dilution parameters 
giving the largest critical capacity is obtained. 

The rest of the paper is organized as follows. In Sec- 
tion[5]the model with variable dilution is introduced from 
a dynamic point of view. Section [21 presents the replica- 
symmetric mean-field calculation of the free energy and 
obtains the fixed-point equations for the relevant order 
parameters. The possible appearance and properties of 
two-cycles are examined. In Section 0] these equations are 
studied in detail for arbitrary temperatures and differ- 
ent types of dilution. In particular, capacity-temperature 
phase diagrams are obtained and compared with the cor- 
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responding results for sequential updating. The optimal 
dilution parameters leading to the largest critical capac- 
ity are obtained. Also the specific thermodynamic proper- 
ties are discussed. Furthermore, the effect of self-coupling 
on the retrieval properties and on the phase-diagrams is 
analized. Some concluding remarks are given in Section|5| 



2 The model 

Consider a network of N neurons, rr = {<J\, crjv}, which 
can take values from the set S = { — 1,0, 1}. In this net- 
work we want to store p = aN patterns, {£f }, i = 1, . . . , TV 
and /i = l,...,p. They are supposed to be independent 
identically distributed random variables (iidrv) with re- 
spect to i and /x, drawn from a probability distribution 
given by 

Pr(tf) = ad(l-(tff) + (l-a)5(tf), (1) 
with a the pattern activity, namely 

E((£f) 2 )=«. (2) 

The neurons are updated synchronously according to 
the transition probability 



«ef) 



M\2 



For convenience 



Pr(<7|o-') =n Pl '( CT '' =s£S\tr') 



(3) 



Pr fa =sGS\a')= ^pHM'K)] (4) 
5>xp[-/Msk')] 

with f3 the inverse temperature and e;(s|er) an effective 
single site energy function given by 

e.;(s|er) = —hi,i(tr)s - h 2 ,i{(r)s 2 , s£S, (5) 

where the random local fields are defined by 

N 



(6) 



In the sequel the index k will always take the values k 
1,2. The couplings J£ ~ are taken to be of the form 



_ c k,ij T 

k.ij ~ J k,i] 
Ck 



(7) 



where the probability distribution of the {ck ij} is given 
by 

Pr(cfc,ij) = c k 5(ck,ij - 1) + (1 - c k )8{cksj) ■ (8) 

Hence, they allow for a diluted architecture. The J k ,ij are 
determined via the Hcbb rule 



1 P 



(9) 



where rj^ i = %g and vfci 

we define a x = E(« 4 ) 2 ') = a, a 2 = E((r/^) 2 ) = a(l - a). 

As can be seen from Q-©, two types of dilution are 
possible. Some of the couplings related to the bilinear term 
in the neurons, k = 1, can be cut and/or some of the 
couplings related to the biquadratic term, k = 2, can be 
pruned, both simultaneously or independently. In the fol- 
lowing we denote this as /c-type dilution. We remark that 
in the literature k = 2-type dilution is frequently indi- 
cated as dilution of the active patterns, since only the 
fact whether the neuron is active or not (and not the sign 
of the neuron state) is important. In order for the model 
to satisfy detailed balance, the dilution has to be symmet- 
ric (c/^jj = Cfcji). In the case of extreme dilution, when 
c k = 0, the average number of connections per neuron, 
c k N is still taken to be infinite, as usual. This can be 
achieved by requiring that 



lim c k N = \ogN . 

c fc ^0 



(10) 



Finally, we recall that the detailed balance property for 
synchronous updating is not destroyed by the presence of 
self-couplings, i.e., couplings of the form J kii , k = 1,2. 
Hence, we do allow for this type of couplings and write 
them as J kii = Jk.o a k/a k , with J k) o a parameter and 
a k = p/c k N the fc-type capacities. 

The long-time behaviour of this model is governed by 
the pseudo-Hamiltonian ^ 



1 N 

H(tr) = ln{2exp (/?/i 2 .i(<r)) cosh (/3/i m ((t)) + 1} 

(11) 

with as duplicate-neuron (spin) representation 



k i,j 



(12) 



such that Tr [cxp(- (3 H(cr))} = TrTr [cxp(-PH(cr : r))l. 
<j er t 

In the next Sections we study the thermodynamic and 

retrieval properties of this model starting from the free 

energy. 



3 Replica mean-field theory 



3.1 Replicated free energy 



We adapt the standard replica technique as applied to 
dilute models ^21" EH to synchronous updating in order 
to calculate the free energy of the model. Since the method 
is rather standard by now, although the specific details are 
rather involved, we only indicate the main steps. For the 
completely detailed calculation we refer to |17| . 
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We start from the replicated partition function based 
upon the two-spin Hamiltonian (|12J) 

P fe,i \ P / 

(nn^pf^E^p)) as) 

\ k \ p / I £ 

where the average is over the patterns and the dilution and 
where we have explicitly separated off the self-interactions. 

We first average over the dilution by expanding the ex- 
ponential with respect to Jf, ^ up to order 0((cfc^V)~ 3 / 2 ). 

After using that the couplings J^jj ~ 0{{akCk / a\N) x / 2 ) 
and assuming a macroscopic overlap with only one pattern 
we arrive at 

life) Z C (Z" C ) 5 



n^exp(^(J M -c fc )E^ 



x exp 



4a 2 k N 



(7 ■ T 

3,P hP 



P 



(14) 



where Z c denotes the condensed part (e.g., pattern /i = 1) 
and Z nc the non-condensed part (patterns /i = 2, ...,p) 

z c (z n % = (n ex p (>e j ^<p4 p ))) ■ 

Next, we consider the average over the disorder. A 
Hubbard-Stratonovich transformation allows us to write 
the condensed part, introducing N /3m\ k , with fc, k' = 
1, 2 (we forget about the upperindex 1 in the sequel), as 



I**/ exp 



2 / j ' ' "kk' ,p 
P 



Z c oc / I| ( dm 

x I1 cx p 1 — E E [vkA°i, P + T t P ) m ki,p 
k \ ak P i 



+iVk,i{<7i, P - Ti, p )mk2,p] (16) 



with dm.kk' = rip m fcfe',p- F° r t ne non-condensed part 
we first do a Hubbard-Stratonovich transformation in- 
troducing the variables rn kk , for fj, > 1, expand then 

the exponential up to order C(A^ -3 / 2 ), and finally use 
cos!i(.t) ~ exp(x 2 /2) to obtain 



^no^V(l-P) a 



y exp (^-^* 2 + ^*** f ) , (17) 



where $ is a 4n-dimensional vector of the form $ = 
($i, * 2 , -, * n ) with * p = (mii, p , mi 2 ,p, m 2 i. p , m 22 ,p). In 
the same way, ^ is a 4rt x 4n matrix of the form 



*n i*i 2 
i*i 3 -*14 

*21 «*22 

i*23 -*24, 



Each element in itself is an n x n matrix with elements 
defined in terms of the neuron variables as 



1 N 

^i,P7 = EKp + T t)^li + T ti) > 

^ ' 1—1 

1 * 

V'fc2,p 7 = E(^P + ^P^tl ~ T t~f) ' 



i=l 
N 



^3,p 7 = EKp ~ T 4 fe p)(^ 7 + T^) , 

i—1 

i N k 

"0fc4,p 7 = 2jVa E(°*> ~~ T i,p)( a t~f ~ T tl) ■ 

The integral over the $ in l|17(l can be done, yielding 
(Z nc ) s excxp ^-y^a fc c fe Tr(ln[I-/3* fcl ])j 
xexp f-y ^a fe c fe Tr(ln [I + /3* fc4 

+/3 2 * fc3 (I-/3*fcir 1, 4 f fc2] ) j . (18) 



At this point we introduce the relevant Edwards- 
Anderson (EA) order parameters 



— 1 \ " „k k 
b^ ~ jy L^ a i,P U i> 

i=l 
1 N 

i=l 
1 " 

v E °t 



k 



jy " hp 'hi 
i=l 



(19) 
(20) 
(21) 



and we redefine the rrikk',p for technical reasons (recall, 
e.g., cci. l|16(0 such that the r and the a can be considered 
separately 



mkk', P = mki, P + (-l) fc 1 imk2, P 



(22) 



Using its self-averaging property in the thermodynamic 
limit, the replicated free energy can then be written in 
terms of these redefined overlaps, the EA order paramc- 
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ters and their conjugate variables as 



is defined as 



/« = w lim-^ log 



mki, p mk2,p - ^2 — (Jk,o - Cfc) r fc , pp 



k p 



(Ik 



^a feCfc Tr(ln[£ fe ]) 



2/3 
P 



X ak X! ( 2r fc,P7^,P7 + X! Qkk'^-yQkk' ,p 7 
fc p,7 fc' 

/3 \ ^ a fc (l - c fc ) ^ / 

o / j 2 ? ' \ ff fcl .P7 ( ?fc2,p7 + 7"fc,p7?"fc,7p 

fc K p,7 



/3 



log 



nflrTr) exp (pH{a,f) 



(23) 



with (j = {ci, . . . , <J n } denoting a vector in replica space 
and 

Sfc = (I-/3¥ fcl )(I + /3¥ M ) 

+ /? 2 (I - /3¥ fc i)¥ fc3 (I - /J^fei)" 1 *^ ■ (24) 

Here H(a,f) is the effective single-site replicated Hamil- 
tonian 

#(^> f ) = X) ( ~ ??fc H( CT p mfcl -p + T P m ki.p) 
k \ ak P 



Both ST^ ( n ~ kk , ~ kk , ~ ^kk\ 

H — 7T- 2^ (2r fciP7 cr p T 7 + g fe i, P7 er p cr 7 + gfc 2 , P7 T p T 7 J 



(25) 



We remark that the ty k i can be expressed in terms of 
the order parameters defined in (|f 9fl - 121i 



la k 



*fcr = ;n~-(qfci +r k +r[ + q fe2 ) 

*fc2 
*fc3 



A'4 



— (qfci - r fe + rj, - q fe2 ) 
2^(qfci+rfc-r^qfc 2 ) 

1 ( t A 

^ — (qfci - r fe -r! + qfea) 



(26) 
(27) 
(28) 
(29) 



The physical meaning of the parameters becomes clear 
when writing 



mfci 



1 

«fc 
1 



% <-p fe ) 



<7fc2„ 



(0*0*^)^(30) 
(<^7%L (31) 



rfc,p 7 = ( 



(32) 



where (-)^ c denotes the average over the condensed pattern 
and initial conditions and where the thermal average (-)o 



n(TrTr) A(a, f) exp (/3i? (a, f)) 
nfTrTr)exp (pH (*,?)) 



(33) 



Next, we want to study the replicated free energy and 
its minima within, as usual, the replica-symmetric approx- 
imation in order to obtain the phase diagrams for the 
model. To make this picture complete we also have to 
address the occurrence of two-cyles as possible stationary 
solutions of the relevant saddle-point equations. 



3.2 Replica symmetry and non-cycle ansatz 

In 0] it has been stated, within a replica-symmetric treat- 
ment, that in the Little-Hopficld model f |18H9j ') cycles do 
not occur in the replica-symmetric retrieval phase. In that 
work the authors assume, however, that the order param- 
eter rfc, P7 (recall (|21|l ') is symmetric, implying already ab- 
sence of two-cycles in the retrieval phase right from the 
start. Extensive simulations for the Little-Hopfield model 
have shown ^U] that, at least in the retrieval phase, cycles 
do not occur. 

We have performed extensive simulations for the fully 
connected BEG neural network with equal dilution pa- 
rameters (implying u\ = a 2 = a) for uniform patterns 
(a = 2/3) with synchronous updating at zero temperature. 
The basic quantity we have studied is the cycle parameter 
A(t), defined as 



1 N 



(34) 



The results are shown in Fig. ^ In particular, we have 
run simulations for several values of the capacity a at 
zero temperature during 1000 time steps of synchronous 
updating, which in all cases reached stationarity (i.e. a 
constant value A{t) > 0). We have determined how many 
of the runs (200 in total) end up in a cycle and how many 
in a fixed-point. Moreover, we have computed the average 
value of A(t) for the cycles. In addition, we have checked 
what is the average number of neurons involved in the 
cycles. To have an idea of the finite size effects, the sim- 
ulations are performed for TV = 4000 up to N = 7000 
neurons. From the results it seems natural to conjecture 
that in the limit N — > oo no cycles will exist in the re- 
trieval region (a ^ 0.091 for T = 8 ). In Fig. 1 we show 
results for the initial conditions mfc(0) = 0.6, go(0) = 0.7 
but simulations for several other initial conditions (even 
for mfc(0) = 0) lead to the same conclusion. 

The first order transition from retrieval to spin-glass 
behaviour includes as well a transition from a region where 
two-period cycles are not occurring to a region in which 
they basically dominate the space of solutions, as can be 
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Fig. 1. Simulations with N = 4000 (squares), N = 5000 
(diamonds), N = 6000 (triangles) and N = 7000 (circles) for 
T = and m fe (0) = 0.6, g (0) = 0.7. From top to bottom, 
we show the average value of A(t) for the runs that reached a 
cycle, the percentage of cycles among the 200 runs performed, 
and the average percentage of neurons flipping in the cycles. 
Only the results for the largest system size have been plotted 
with a full line. 



concluded from the figure showing the percentage of cy- 
cles. 

Furthermore, we see that the cycles never involve a 
large number of neurons (always less than 0.5% of the to- 
tal in the figures). The average number of flipping neurons 
increases monotonically with a but does seem to saturate 
with increasing system size. This is in agreement with the 
results obtained in a finite-size scaling study of the capac- 



ity problem for the Hopfield model |20] where cycles also 
involve only a tiny fraction of all neurons. Moreover, these 
authors find that there is always a fixed-point solution of 
the sequential updating nearby. In this sense, cycles seem 
to be stationary solutions in which the system is unable to 
reach an accessible spin-glass minimum but jumps around 
it. 

Given these findings, we neglect cycles in the sequel 
of the replica calculation since we are interested mainly 
in the retrieval phase. In order to implement this non- 
cycle ansatz, we eliminate any distinction between a and 
r. Therefore, we introduce the following assumptions 



mki.p = m k2 ,p = m kiP 

9fcl,P7 = <7fc2,p7 = <7fc,p7 I <7fcl,7 = 

rk !fn = Qk.tn , only for p ^ 7 . 



9fc2,p 7 



(35) 
q k , P7 (36) 
(37) 



We explicitly remark that introducing r ktPP = qk.pp would 
not be correct since both quantities imply products of 
spins from the same replica, but a and r are affected by 
different thermal fluctuations. 

Finally, we impose symmetry in the replica space. Be- 
fore doing so we solve for the conjugate variables in order 
to further simplify the free energy. This allows us to re- 
place rk, PJ by <Zfc,p7 for p ^ 7. The replica symmetry (RS) 
ansatz then implies 



m k , P 
q~k, P ~/ 

fk,pp 



m k 
Qk , 

rk.o 



Qk, PP = <Zo 

q~k,pp = <Zfc,0 

fk, PP = h,o ■ 



(38) 
(39) 
(40) 
(41) 



As a result, the free energy per spin in the limit n — > 
can be written as 



lim — = 1 

n->0 n 



2/3 



k 

ak 



- c k )r kfl 
Cfc ' 



Xkrf 



xt] 



Wqk 



Ofc(l - Xkr ~ Xk) 



^ a k 



_P_ \ - Q!fc(l 

9 2^ , 



'k.O 











{ql - H 



Vz{Dz 2 log 



TrTr exp 



(f3H(a,T;z k ))]^ (42) 



with z\ and z 2 Gaussian variables with measure T>z = 
dz(27r) -1//2 exp(— z 2 /2). The effective Hamiltonian now reads 



r k ) +a k a k Xkr<y k T k 



H(a,T;z k ) = y~] (—Vkm k (a 
k Kak 

+ \{< j2 + T 2 )a k a k Xk + V a kq k {cr k + r fc )z fe ) (43) 
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with the implicit definitions for the susceptibilities 

Xk = — (qo - qk) , Xkr = —(rk.o - qk) (44) 

a k a k 

P P 
Xk = — (q~k,o ~ qk) , Xkr = — (rk,o - qk) ■ (45) 
a k a k 

We remark that by setting a = 1, as well as J 2y o = 
^2,0 = 92 = r 2:0 = q 2 = 0, q 2 fi = <?i,o = and q a = 1 
we exactly recover the free energy for the Hopficld model 
with synchronous updating and variable dilution 



3.3 Saddle-point equations 

The extremization of the free energy in O yields 

m k = ^{v k (a%) iez , (46) 

* - (<a>^ > * - (<**>;)«.„ ■ ^ 

r M = ((aV%) , (48) 



with (•) the average over the Gaussian noise (recall the 
last term in l|42|l) and (-)g the thermal average performed 
with respect to (031 . The conjugate variables satisfy 



q k . 

Ik = -2 1 1 - Ck 

u k 

Xk , 

Xk = — [1- c k 



c k 



(1 - Xkr ~ Xk) 2 

Cfe 

(1-Xfer) 2 -Xfe 



(49) 
(50) 



1 f n \ Cfc(l-Xfcr) . T 

Xkr = ~n (1 - C k )Xkr + JZ J + J *fl ~ C *-' 

«fe V (1 - XkrY - XI 



(51) 



In the absence of thermal noise (T = 0), the system 
reaches a minimum of the free energy such that <x; = Tj, 
Vi. Therefore, it is interesting to consider the limit a = r 
of the free energy and the saddle-point equations. The 
trace of the effective partition function (recall 143(1 ') has to 
be evaluated again and, in addition, the following equiva- 
lences hold 



For the saddle-point equations at T — we then ob- 
tain, with obvious notation 

m k = — (wngn^ifo)) e(|&i(*0| + h 2 (z 2 ))) 
a k \ 

(54) 
(55) 



q = (e{\h 1 {zi)\ + h 2 {z2)) 



Xk 



a k \/a k q k 

x (zkSign^h^)) 0(11^)1 +h 2 (z 2 ))) 



where the effective fields are given by 

2 

_] ay \a k >Xk' + 



ftfc(Zfc) = —Vk m k + 4.2 V* ( afe'Xfe' "+ 



+ \JoL k q k z k . 
The conjugated variables read 



«7 



9fe = — I 1 - Ck 



(l-Xk) 2 J ' 



Xfe /-, , c k 

Xk = — 1 - c fc 



1 - Xfc 



(57) 



(58) 



(59) 



4 Thermodynamic and retrieval properties 

In this Section we study the thermodynamic and retrieval 
properties of the BEG network with variable dilution and 
synchronous updating by numerically solving the saddle- 
point equations (|46[) - l|51[) an d compare the results with 
those for sequential updating. As in the model with se- 
quential updating there are four phases: the retrieval 
phase R (m k > 0, qi > 0), the spin-glass phase SG 
(m k = 0, q\ > 0), the paramagnetic phase P (m k = 0, 
cji = 0) and the quadrupolar phase Q (mi = 0, m 2 > 0, 
qi > 0). In the sequel we discuss the phase diagrams for 
low loading (a k = 0), finite loading (ak > 0), fc-type dilu- 
tion and the effect of the self-couplings J k $. 



4.1 Low loading 



rk,o = qo , Xkr = Xk , (52) 

1 . 

Xkr = Xk, r kfi = q k ,o-\ „J k ,0 ■ (53) 

a k P 

For zero self-couplings (Jk,o = 0), this leads to the re- 
lation Pf S yn(P) = 2Pfseq(2P), as is also the case for the 
Q = 3 Ising network ^U] . This is easy to understand when 
noticing that for r* = <Tj the Hamiltonian l|12|) for syn- 
chronous updating becomes simply twice the one for se- 
quential updating. An immediate consequence is that the 
saddle-point equations are identical to those for sequential 
updating [8TfT)j . 



For a k = 0, the saddle-point equations can be derived 
without the use of replicas (e.g., |21I22| for the Hopficld 
model) and we find 

1 I Trer fc cxp f^Efe' ^Vk'm k ', T o- k ') \ 

m k , a = — ( Vk— 7 ) 

a k \ Tr exp [pY.w TzWm^r**) / ^ 

f (60) 

and similarly for vn k r . One can show that the only mini- 
mum of the free energy is given by m k tT = m k T , such that 
the solution coincides which the one for sequential updat- 
ing. Physically, this means that in equilibrium there can 
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Fig. 2. The a — T phase diagram for the fully connected 
BEG neural network with activities a — 0.5 ,2/3 and 0.8 for 
synchronous and sequential updating (thick respectively thin 
lines) . 



Fig. 3. The a — T phase diagram for the extremely diluted 
BEG neural network with activities a = 0.5, 2/3 and 0.8. 



4.2 Finite loading 



be no cycles: the two neuron variables have decoupled be- 
coming independent and leading to 0f syn (0) = 2/3/ seg (/3), 
as seen in the other models mentioned above |1I1UI21|22] . 
Hence, the dynamics docs not influence the equilibrium so- 
lutions and the a — T phase diagram is identical to the one 
for sequential updating. Furthermore, we know jS] that for 
low loading the form of the architecture (fully connected, 
extremely diluted) is not important. Hence we do not treat 
this case in more detail here but we refer to, e.g., [S] (Fig- 
ure 1) and the discussion therein. 



For the fully connected architecture, c k = \, Fig.Opresents 
the capacity-temperature phase diagrams for three typical 
values of the activity a = 0.5, 2/3 (uniformly distributed 
patterns) and 0.8. For the moment, the self-couplings are 
set to zero and the dilution parameters arc equal, i.e., 
c i = C2 (recall that in that case a k = a). Dashed lines 
correspond to second order transitions, full lines to first 
order transitions and dotted lines to thermodynamic tran- 
sitions. The thick lines indicate the results for synchronous 
updating; for comparison we have indicated those for se- 
quential updating [H] with thin lines. 
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As can be seen the main features for both types of 
updating are very similar. For synchronous updating the 
spin glass region increases visibly while the retrieval re- 
gion increases marginally. A similar behaviour is observed 
for the Hopficld model |4TTU| . Finally, for a = 0.8, the 
quadrupolar region is slightly enlarged as well. 

For extreme symmetric dilution (ck = 0) the phase 
diagrams nearly coincide with the ones for sequential up- 
dating ^Uj. Since both sets of results did not yet appear 
in the literature we show them in Fig. [3J 

Compared with the fully connected architecture, most 
of the first order transitions become second order. Only 
the quadrupolar-paramagnct first order transition remains. 
Furthermore, there is strong re-entrant behaviour in the 
retrieval region. This is related to the RS approximation 
and is also seen in the Hopfield model |12ll3j and the 
Q = 3-Ising model 



4.3 k-type dilution 

Next, we look at the effect of fc-type dilution on the system 
(recently introduced as asynchronous dilution in a Gard- 
ner optimal capacity calculation Indeed, the BEG 
model allows one to prune independently the two types of 
couplings appearing in the Hamiltonian (1121) . 

The results can be summarized as follows. The perfor- 
mance of the network strongly depends on the activity. In 
any case, the absolute capacity p/N decreases when the 
pruning becomes stronger, as one expects, while the stan- 
dard capacity p/cN shows a maximum. 

Hence, given the number of connections per neuron, 
c = ci + C2, there has to be an optimal combination of 
both dilution variables ci and C2- In order to be able to 
predict this combination once the other parameters of the 
system arc known we have performed some numerical ex- 
periments. We present in Fig.0]thc critical capacity a c for 
the activities a = 0.5, 2/3 and 0.8 as a function of c\. The 
values for c are c = 0.5, 0.75, 1.0, 1.25 and 1.5, from top 
to bottom, and the temperature is fixed at T = 0.5. We 
find a maximum in all figures, corresponding to the opti- 
mal combination of the dilution parameters c = cj + c* 2 . 
Looking, e.g., at the optimal value c\ as a function of the 
number of connections per neuron c for T = 0.5 for several 
values of the activity, a = 0.9, 0.8, 2/3, 0.5 and 0.4 we find 
that these data points can be fitted by 

c\ = (l-a)(ci + c 2 ) = (l-a)c. (61) 

We conjecture that the temperature does not modify the 
position of the maximum but only its height, since c\ is 
determined by the role of both terms in the Hamiltonian 
and therefore by the activity. 

Furthermore, we notice a second maximum for a = 0.8. 
This is due to the existence of a second retrieval solution 
occurring for large activities [S]. This solution only ap- 
pears for < c < 1.4 (for T = 0.5). We can fit this second 
peak by 

cl = a(ci + C2) = ac . (62) 





Cl 




0.2 0.4 0.6 0.8 1 

Cl 

Fig. 4. Standard critical capacity at T = 0.5 as a function of 
ci. From top to bottom, a = 0.5, 2/3 and 0.8. In each figure, 
from top to bottom, c = 0.5, 0.75, 1.0, 1.25 and 1.5. 



Since for larger activities this relation implies that there 
are fewer fc = 2-type connections at the maximum c{ , the 
second retrieval solution has a smaller overlap m 2 . 

Furthermore, when ci is kept finite, both critical ca- 
pacities for C2 = are always finite. However, when C2 is 
kept finite, setting c\ = destroys any retrieval. This sug- 
gests that it is actually the first term of the Hamiltonian 
that mainly ensures the process of retrieval. Keeping only 
the second term of the Hamiltonian, the system still rec- 
ognizes correctly the zero and the active states, but does 
not discern between the +1 and —1 states. It still im- 
plies though that the information content of the system is 
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nonzero. A practical realization of this behaviour may be 
in pattern recognition where, looking at black and white 
pictures on a grey background, the system would tell us 
the exact location of the picture with respect to the back- 
ground without finding the details of the picture itself. 
These findings seem to be in agreement with the results 
in |23] for the Gardner optimal capacity in the presence 
of dilution, where it is concluded that the BEG network 
is indeed more robust against k = 2-type dilution of the 
couplings. 



4.4 The self-couplings effect 

Finally, we turn to the study of the effect of self-coupling 
on the phase diagrams of the model. In the case of very 
large but finite self-couplings Ji.o and/or J2,o> the er and 
r tend to stay in the same state. Then, the Hamiltonian 
becomes 



lim H syn (a) 



Jk,a»l 



2H, 



seq 



a k 

fc=l 



N 
i=l 



(63) 



The last term favours the elimination of the zero states 
from the system as can also be seen from eqs.(@}-(|5j) in 
these limits. The order parameters of the system evolve 
to mi = 0, qo = 1, 92 = 1, r 2 ,o = 1 and 

(i) for Ji o — ► 00 or Ji : o and J2,a —> 00 

'm tanh(2j9fti(«i)) 



mi 



1 



qi = (tanh 2 (2/3h 1 (z 1 )) 



r i,o 



(64) 

(65) 
(66) 



(ii) for J 2i o — > 00 



when o = l, and the Hamiltonian reduces to a Hopficld- 
like one with a rescaled (3 and at = t, for any temperature. 
In the second case erf = r 2 for any temperature, but only 
for T = we are ensured that cr, = r, as well. 

In Fig. [5] some phase diagrams in the presence of self- 
couplings Jk.o = ak ar e shown in comparison with the 
results without self-couplings. For the fully connected ar- 



1.4 
1.2 
1 

0.8 
0.6 
0.4 
0.2 
0, 



P ,„,--r- 






SG 


R 









0.03 



0.06 

5 



0.09 



0.12 




a 



Fig. 5. The a — T phase diagrams for a = 2/3 with Jk.o = 
(thin lines) and Jk.o = dk (thick lines) for the fully connected 
(top) and the extremely diluted (bottom) BEG networks. 



1 / sinh(2/3/i 1 (^ 1 )) 
mi = — ( m = — 

ffll \ COSh(2/3/li(zi)) + e -2/3«iaiXlr 



sinh 2 (2/J/ii (21 ) 



\ cosh 2 (2/3/ii (21)) + e - 4 ^ Q i a ixir 

/ COSh (2/3/11 (Zi)) - e -2/3aiaiXl, 
\ COSh (2/3hx(zx)) + e -2/3aiaiXl. 



(67) 
(68) 

(69) 



with hi(zi) given by l|57() (fc = 1). As can be seen from 
these equations, in the first limiting case we arrive at a 
Hopfield model with sequential updating when a — > 1, 
with the usual rescaling in j3 (/3 — ► /3/2). For the second 
limiting case, we only evolve to the Hopfield model with 
sequential updating in the limit /3 — > 00, a — > 1. Indeed, in 
the first case, the second local field /12 becomes irrelevant 



chitecture, c k = 1, there is little difference between syn- 
chronous and sequential updating. While the spin glass re- 
gion is visibly enlarged, the retrieval region has increased 
only marginally. 

The extremely diluted network is much more sensitive 
to the presence of self-couplings. The retrieval region is 
substantially enlarged, as well as the spin glass region. A 
new feature is the different re-entrance point for different 
values of the self-coupling. This is not observed in the Hop- 
field model ^H] because the limit of large self-couplings 
in that case is the Hopfield model for sequential updat- 
ing, which is known to have the same RS critical capacity. 
Here the phase diagrams must evolve from the BEG phase 
diagram with synchronous updating to the Hopfield one 
with sequential updating, which has a different critical ca- 
pacity. Therefore, the re-entrance point also evolves as we 
increase Jk,o- A similar effect has been observed for the 
Q-Ising model with finite dilution (c = 0.01, |10j). 
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Hence, the self-couplings in three-state neural networks 
with variable dilution are very relevant. Different values 
can produce families of phase diagrams that evolve to- 
wards the Hopficld phase diagram for sequential updat- 
ing with the rescaling in f3 mentioned before. An illus- 
tration of this effect is presented, for T = 0, in Fig. 
where the critical capacity is shown for different values 
of the self-couplings as a function of the activity. We re- 




0.2 0.4 0.6 0.8 1 

a 



Fig. 6. Critical capacity as a function of the activity a for T = 
and different values of the self-couplings Jk,o = (circles), 
10 (diamonds), 100 (triangles up) and oo (triangles down). 



mark that for a = 1 we recover the value for the Hopfield 
model, a c ~ 0.139. A maximum in the critical capacity 
appears unless the self-couplings are large enough so that 
the three-state nature of the model is breaking down since 
all zero states are completely eliminated. 



5 Concluding remarks 

We have considered the thermodynamic and retrieval prop- 
erties of BEG networks with synchronous updating and 
variable dilution. Saddle-point equations for the relevant 
order parameters have been derived for arbitrary temper- 
ature using a replica mean- field treatment. In the station- 
ary limit, the system allows for fixed-points as well as two- 
cycles. It is found numerically that two-cycles only occur 
in the spin-glass phase of the system, never involve a large 
number of neurons and are always close to a fixed-point 
spin glass solution. 

Making a non-cycle and replica symmetry ansatz, the 
phase diagrams of the model are obtained. Two types of 
dilution involving the two types of couplings arc discussed. 
The optimal combination of the network dilution param- 
eters giving the largest critical capacity is obtained. It is 
also found that the model is more robust against dilution 
of the k = 2-type couplings. 

The results for different pattern activities and variable 
dilution are compared with those for sequential updating. 
It is observed that the asymptotic behaviour of the net- 
work with synchronous and sequential updating is almost 



identical. For some values of the network parameters the 
retrieval region is enhanced marginally. Only the presence 
of self-couplings can enlarge the retrieval region substan- 
tially, especially in the case of extreme dilution. 
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